Using Genetic Algorithm Based Variable Selection to Improve Neural Network Models for Real-World Systems

Real-world systems are often modeled by sampling sensor data taken during system operation. System states may not be all known or measurable, sensor data may be biased or noisy, and it is not often known which sensor data may be useful for predictive modeling. Neural network models generated from this data must therefore rely on how effectively the chosen sensor data represents the system. Genetic algorithms may help to address this problem by determining a near optimal subset of sensor variables most appropriate to produce good models. This paper describes the use of genetic algorithms to optimize variable selection to determine inputs into a neural network system model. The use of this technique for modeling a typical industrial application, a liquid fed ceramic melter, and the results of the genetic search to optimize the neural network model for this application, are described. Keywords/ neural network, genetic algorithm, optimization, variable selection, feature selection 1 This research was supported by the Office of Naval Research work requests N0001402WR20090 and N0001402WX20003. 0ntroduction When modeling a complex system (such as a chemical reactor), it is not generally known a priori which system states are necessary to develop a good model, or which states are observable based upon available sensor technology (although it is often known that many system states are not observable). In addition, there is a greater problem in identifying useful data. Complex dynamic systems such as the chemical reactor may be instrumented with tens, hundreds or even thousands of sensors. The problem with so much sensor information is that most of it will be irrelevant. Worse still, unfiltered incorporation of irrelevant data will adulterate a model, eroding its predictive capabilities. A key data pretreatment problem is sensor redundancy. It is well known that smaller models are often better models [5], [4]. This translates to fewer inputs and fewer hidden layer nodes. While it may be nice to have highly redundant data from a large number of sensors, in reality we may only need a few key sensors in order to produce a good model. The problem is in determining which few sensors to choose, and ignoring most of the remaining sensors. This is confounded by the fact that due to differing sensor response characteristics and noise, in the aggregate there is a considerable amount of noise and bias in the data. In this study modeling of a liquid fed ceramic melter (LFCM) process was undertaken in order to predict the surface level. The melt chamber was instrumented with 20 thermocouple sensors placed at different sites within the chamber. Each sensor may have a slightly different characteristic response curve due to differences in manufacturing, usage history, etc. Each sensor also is susceptible to some level of noise. A time history of data is taken from all 20 sensors and stored in a database used to train a neural network model. Some sensors, such as those near the surface in the reactor vessel, may offer fairly high-variance data throughout the process, but be largely irrelevant to accurately predicting final product quality. We would like to select a near-optimal set of sensor variables in order to train a neural network model with the greatest predictive accuracy 5ariable Selection 8sing :As Variable selection (or feature selection) may be performed in an automated way using genetic algorithms (GAs). The genes need to be defined for a given application such that finding a better or more optimal set of genes means finding a better solution to the problem. A GA may perform variable selection if each gene in a chromosome represents an available sensor variable. Fitness is judged for each chromosome by determining how good the models are (accuracy, robustness) generated by that combination of variables. An initial population of chromosomes is generated by choosing a string length (# of genes) and randomly assigning a variable to each gene. The GA search is then set in motion and the chromosomes compete, reproduce, and die off as they are replaced by more fit chromosomes. It is usually desirable to maintain a fixed-size population in order to make sure that the fitter chromosomes quickly replace the less fit ones. An occasional mutation is introduced to make sure that certain genes (variables) which may be really useful aren't quickly eliminated (possibly because they are randomly combined with really noisy variables early on) and then never incorporated again. This is referred to as a population in danger due to lack of genetic variation, and to avoid this situation a mutation rate is predetermined and mutated chromosomes are introduced into the population at regular intervals during GA search. As these parameters are application dependent, it is not possible to know beforehand which values will work best. The GA process is implemented with automatic sequence selection, model building and discarding, and evaluation of accuracy and robustness of the models (scoring). Successive generations will inherit the best characteristics from the previous generation, while eliminating the less valuable characteristics. :A ;epresentation = >perators Genetic algorithms are often thought of, discussed and implemented using binary strings, or bit strings. Each gene or bit represents the expression of a state. If the bit is turned on, then the gene corresponding to that bit can be said to be "expressed". In this application a bit represents the state of either a variable being included (“1”) or not included (“0”) in the final solution. Genetic algorithms sometimes require the use of special operators in order to simulate the evolutionary processes which they emulate. The most common operators are crossover and mutation. The crossover operator takes two parent chromosomes (in this application, each parent chromosome represents a group of input variables used to build a neural network model), and combines them to produce an offspring. A common form of crossover operator is uniform crossover [6]. In uniform crossover, if a specific gene is turned on in both parents, then it will be turned on in the offspring. If a gene is turned on in only one of the parents, then it may be turned on (with a predetermined probability, usually 0.5) in the offspring. Uniform crossover was used in this project. The mutation operator is applied independently but immediately following the crossover operator. A mutation is a random change of a gene in a chromosome, and is applied according to a preset mutation rate (usually quite low, e.g. 0.001). An elitist policy, or survival rate, that determines what percentage of the population (the fittest members) would survive into the next generation was employed. Because the computational cost of building and training neural network models from scratch can be high, another feature employed in this work was to guarantee that when a new offspring is generated it does not duplicate any chromosome currently in the population or which has been previously built and tested. A graveyard was used to store old chromosomes which represent models which have been built, tested, and then discarded. Each new offspring is compared with chromosomes in the graveyard to make sure that it hasn't been tested before in a previous generation. Since we assume that all of the neural network models use the same superset of data (same output data, input data includes sensor streams for all possible input variables), then the process of choosing variables for a particular model is deterministic, so there is never a need to retest a chromosome once its corresponding model has been built and scored. This promotes better crossover by preventing the generation of chromosomes which are already represented or have been generated and tested in prior generations. Chromosomes which are carried from one generation to the next are stored along with their scores, but are not retested since this would unnecessarily duplicate computations.